Adding more scenario-based tests by FabianMeiswinkel · Pull Request #49568 · Azure/azure-sdk-for-java

FabianMeiswinkel · 2026-06-19T08:53:57Z

Description

This PR adds scenario-based tests - the original test-suite was provided by a customer who used these for release valdiations. Below is an analysis of the tests and triage of gap analysis and brief summary what additional coverage these tests provide.

Customer test file / area	Customer coverage intent	Initial SDK coverage / initial gap triage	New workflow coverage added	New test location
`test/CosmosDriverDynamicRequestOptionTest.java`	Validate dynamic per-operation request options and diagnostics request-option propagation across app-style operations.	SDK already had broad primitive coverage in `OperationPoliciesTest`; initial triage: `partial` workflow gap.	Added customer-style chained create/read/query/readMany/upsert/delete coverage with excluded regions, keyword identifiers, consistency/read-consistency options, and diagnostics request-option assertions.	`azure-cosmos-tests/src/test/java/com/azure/cosmos/workflows/customer/CustomerWorkflowRequestOptionsTest.java`
`test/CosmosMultiFeatureTests.java`	Validate app-style create/read/query/upsert/delete, custom correlation IDs, invalid readMany session token, and regional routing.	SDK had primitive coverage in `CosmosDiagnosticsTest`, `ExcludeRegionTests`, and `SessionConsistencyWithRegionScopingTests`; initial triage: `partial`.	Added keyword identifier propagation across operations, excluded-region diagnostics, CRUD-like workflow coverage, and invalid advanced session-token readMany behavior.	`CustomerWorkflowRequestOptionsTest.java`, `CustomerWorkflowSessionTokenTest.java`
`test/CosmosDaoTest.java`	Validate DAO-style CRUD, readAll, patch, transactional batch, bulk operations, custom serializer, and wrapper behavior.	SDK had primitive coverage in `CosmosItemTest`, `DocumentCrudTest`, `CosmosBulkTest`, `CosmosBulkAsyncTest`, `TransactionalBatchTest`, and `CosmosItemSerializerTest`; initial triage: `partial` / `wrapper-specific`.	Added SDK-native DAO-style workflow coverage for CRUD, readAll, patch, transactional batch, bulk read/patch with max micro-batch sizing, excluded regions, keyword identifiers, diagnostics, and request-level serializer propagation. Wrapper cache/session-map defaults remain intentionally out of scope.	`CustomerWorkflowDaoStyleOperationsTest.java`
`test/Latest_Committed_Tests.java`	Validate latest-committed reads with excluded regions, consistency combinations, diagnostics, RU comparisons, and fault behavior.	SDK had primitive coverage in `CosmosLatestCommittedItemTests`, `GatewayReadConsistencyStrategyE2ETest`, and `ClientRetryPolicyE2ETests`; initial triage: `partial`.	Added latest-committed point read, query, readMany, change feed, excluded-region diagnostics, direct/gateway client variants, and regional lease-not-found fault coverage. Strict RU comparison parity remains lower-value/service-sensitive.	`CustomerWorkflowLatestCommittedTest.java`
`regression/direct/*.java`	Validate direct-mode latest-committed regression matrix for change feed, read, query, readMany, session/eventual combinations.	SDK had `CosmosLatestCommittedItemTests` plus change feed tests; initial triage: `partial`.	Added direct TCP latest-committed coverage for read, query, readMany, change feed, excluded regions, and faulted read diagnostics.	`CustomerWorkflowLatestCommittedTest.java`
`regression/gateway/*.java`	Validate gateway latest-committed regression matrix.	SDK had gateway read-consistency coverage in `GatewayReadConsistencyStrategyE2ETest` and spy-wire tests; initial triage: `duplicate` / `partial`.	Expanded latest-committed workflow to run with both direct and gateway session-consistency builders.	`CustomerWorkflowLatestCommittedTest.java`
`test/CosmosHighE2ETimeoutTest.java`	Validate E2E timeout behavior under response delay and partition-migrating faults for create/query/readMany/batch and related workflows.	SDK had primitive coverage in `EndToEndTimeOutValidationTests`, `EndToEndTimeOutWithAvailabilityTest`, and `FaultInjectionWithAvailabilityStrategyTestsBase`; initial triage: `partial`.	Added response-delay workflows with E2E timeout and availability strategy for create, read, query, readMany, upsert, batch, patch, plus partition-migrating read.	`CustomerWorkflowHighE2ETimeoutTest.java`
`test/CosmosStoredProcedureTest.java`	Validate stored procedure create/read/update diagnostics under response delay and read-session-not-available faults.	SDK had stored procedure CRUD/query coverage, but less customer-style fault diagnostics coverage; initial triage: `gap` / `partial`.	Added stored procedure create/read/execute with script logging and metadata fault-rule coverage. Exact stored-procedure-specific response-delay/read-session fault parity remains limited because FI has no stored-procedure-specific operation type.	`CustomerWorkflowStoredProcedureTest.java`
`test/ChangeFeedProcessorTest.java`	Validate CFP start/stop, latest-version handler, current state, restart, and fault-injected read feed.	SDK had broad CFP primitive coverage in incremental/full-fidelity CFP tests and container change feed tests; initial triage: `partial`.	Added latest-version CFP start, restart, current state/lag, and read-feed fault recovery using polling/latches instead of fixed sleeps.	`CustomerWorkflowChangeFeedProcessorTest.java`
`test/PartitionLevelCircuitBreakerTests.java`	Validate PCLB app-chain behavior and query-plan distinction under regional faults.	SDK had PCLB primitive coverage in `PerPartitionCircuitBreakerE2ETests` and `PerPartitionAutomaticFailoverE2ETests`; initial triage: `partial`.	Added PCLB-oriented point read, query-plan diagnostics/query, and patch app-chain workflow under the PCLB-enabled live matrix leg.	`CustomerWorkflowPartitionLevelCircuitBreakerTest.java`
`test/Cosmos429test.java`	Validate 429 and connection-delay behavior in app-shaped calls.	SDK had retry/throttle/fault-injection primitive coverage; initial triage: `duplicate` / `partial`.	Added 429-style rows in multi-write and single-write scenario matrices, plus timeout/response-delay workflows. Connection-delay/connect-reset transport parity remains covered by existing transport/FI suites.	`CustomerWorkflowAvailabilityFaultMatrixTest.java`, `CustomerWorkflowSingleMasterAvailabilityTest.java`, `CustomerWorkflowHighE2ETimeoutTest.java`
`test/CosmosConflictResolutionTest.java`	Validate multi-client conflict detection and conflict query.	SDK had `CosmosConflictsTest`, `ConflictTests`, and `MultiMasterConflictResolutionTest`; initial triage: `duplicate` / `partial`.	No new customer workflow added; existing SDK conflict tests already cover the core SDK behavior. Only exact customer multi-client ordering/diagnostics would remain.	Existing SDK tests only
`multimaster/direct/*.java`	Validate multi-write direct availability strategy matrix across 404/408/410/429/500/503 and read/create/upsert/replace/delete/patch/query/readMany.	SDK had extensive primitive FI coverage in `FaultInjectionWithAvailabilityStrategyTestsBase`, `FITests_*`, and PPAF/PCLB tests; initial triage: `partial`.	Added expanded direct multi-write scenario matrix covering read, query, readMany, create, upsert, replace, delete, patch across representative 404/408/410/429/449/500/503 families.	`CustomerWorkflowAvailabilityFaultMatrixTest.java`
`multimaster/gateway/*.java`	Validate multi-write gateway availability strategy matrix across common server errors and operation types.	SDK had gateway FI primitive coverage in gateway FI suites; initial triage: `partial`.	Expanded availability workflow to run with both direct and gateway session-consistency builders, adding scenario-level gateway coverage without copying each customer class one-for-one.	`CustomerWorkflowAvailabilityFaultMatrixTest.java`
`multimaster/direct/BasicBehaviorMultiMasterTest.java`	Validate basic multi-write account behavior.	SDK already had multi-master and CRUD coverage; initial triage: `duplicate` / `partial`.	Covered indirectly by current CRUD, request-option, latest-committed, session-token, and multi-write fault workflows.	`CustomerWorkflowDaoStyleOperationsTest.java`, `CustomerWorkflowRequestOptionsTest.java`, `CustomerWorkflowLatestCommittedTest.java`, `CustomerWorkflowSessionTokenTest.java`, `CustomerWorkflowAvailabilityFaultMatrixTest.java`
`multimaster/direct/CosmosFeatureValidationMultiMasterTest.java`	Validate customer feature workflows on multi-write accounts.	SDK already had overlapping primitive coverage; initial triage: `duplicate` / `partial`.	Covered indirectly through multi-write workflow suite: request options, latest-committed, session token, E2E timeout, PCLB, and fault matrix workflows.	`azure-cosmos-tests/src/test/java/com/azure/cosmos/workflows/customer/*`
`multimaster/direct/CosmosSessionTokenMismatchMultiMasterTest.java`	Validate session token mismatch behavior on multi-write accounts.	SDK had session consistency and invalid session token primitive coverage; initial triage: mostly `duplicate` / `partial`.	Added valid and advanced readMany session token workflow that validates read-session-not-available behavior.	`CustomerWorkflowSessionTokenTest.java`

All SDK Contribution checklist:

The pull request does not introduce [breaking changes]
CHANGELOG is updated for new features, bug fixes or other significant changes.
I have read the contribution guidelines.

General Guidelines and Best Practices

Title of the pull request is clear and informative.
There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, see this page.

Testing Guidelines

Pull request includes test coverage for the included changes.

Copilot

Pull request overview

This PR adds a new set of Cosmos “customer workflow” scenario-based live tests (fault-injection focused), wires them into Maven profiles/TestNG suites, and updates the Cosmos live test matrix to run these workflows in CI.

Changes:

Added new com.azure.cosmos.workflows.customer.* workflow test suite covering request options, latest-committed, session token behavior, CFP restart, availability fault matrices, circuit breaker, timeouts, stored procedures, and single-master multi-region scenarios.
Introduced new TestNG suite XMLs + Maven profiles (fi-customer-workflows, fi-sm-customer-workflows) to run these workflows independently.
Updated Cosmos live platform matrix and suite setup/shutdown groups to include the new workflow groups.

Show a summary per file

File	Description
sdk/cosmos/live-platform-matrix.json	Adds new CI matrix legs / display names to run the new workflow profiles (including single-master multi-region).
sdk/cosmos/azure-cosmos-tests/src/test/resources/fi-sm-customer-workflows-testng.xml	New TestNG suite for single-master customer workflows group.
sdk/cosmos/azure-cosmos-tests/src/test/resources/fi-customer-workflows-testng.xml	New TestNG suite for multi-master customer workflows group.
sdk/cosmos/azure-cosmos-tests/src/test/java/com/azure/cosmos/workflows/customer/CustomerWorkflowTestBase.java	New shared base providing region discovery, container helpers, diagnostics assertions, and FI rule helpers for workflow tests.
sdk/cosmos/azure-cosmos-tests/src/test/java/com/azure/cosmos/workflows/customer/CustomerWorkflowStoredProcedureTest.java	Adds stored procedure workflow test with script logging and metadata delay FI coverage.
sdk/cosmos/azure-cosmos-tests/src/test/java/com/azure/cosmos/workflows/customer/CustomerWorkflowSingleMasterAvailabilityTest.java	Adds single-master multi-region workflows validating routing/availability behavior under faults and excluded regions.
sdk/cosmos/azure-cosmos-tests/src/test/java/com/azure/cosmos/workflows/customer/CustomerWorkflowSessionTokenTest.java	Adds workflow validating advanced session token behavior for `readMany`.
sdk/cosmos/azure-cosmos-tests/src/test/java/com/azure/cosmos/workflows/customer/CustomerWorkflowRequestOptionsTest.java	Adds chained CRUD/query/readMany workflow validating request option propagation via diagnostics.
sdk/cosmos/azure-cosmos-tests/src/test/java/com/azure/cosmos/workflows/customer/CustomerWorkflowPartitionLevelCircuitBreakerTest.java	Adds PCLB-oriented workflow validating circuit breaker and query plan diagnostics shape.
sdk/cosmos/azure-cosmos-tests/src/test/java/com/azure/cosmos/workflows/customer/CustomerWorkflowLatestCommittedTest.java	Adds latest-committed workflows across read/query/readMany/change feed, plus FI fault coverage.
sdk/cosmos/azure-cosmos-tests/src/test/java/com/azure/cosmos/workflows/customer/CustomerWorkflowHighE2ETimeoutTest.java	Adds response-delay + E2E policy workflows over multiple operation types.
sdk/cosmos/azure-cosmos-tests/src/test/java/com/azure/cosmos/workflows/customer/CustomerWorkflowDaoStyleOperationsTest.java	Adds DAO-style workflow coverage for CRUD/readAll/patch/batch/bulk with diagnostics assertions.
sdk/cosmos/azure-cosmos-tests/src/test/java/com/azure/cosmos/workflows/customer/CustomerWorkflowChangeFeedProcessorTest.java	Adds latest-version CFP workflow with restart and injected read-feed delay, using polling instead of sleeps.
sdk/cosmos/azure-cosmos-tests/src/test/java/com/azure/cosmos/workflows/customer/CustomerWorkflowAvailabilityFaultMatrixTest.java	Adds representative FI scenario matrix across operations/error families with diagnostics assertions.
sdk/cosmos/azure-cosmos-tests/src/test/java/com/azure/cosmos/rx/TestSuiteBase.java	Includes new workflow groups in suite-level setup/shutdown.
sdk/cosmos/azure-cosmos-tests/pom.xml	Adds new Maven profiles to run the new workflow suites via failsafe/TestNG XML.
sdk/cosmos/azure-cosmos-tests/CUSTOMER_WORKFLOW_COVERAGE_MAP.md	Documents coverage mapping and porting rules for the new customer workflow suite.

Copilot's findings

Files reviewed: 17/17 changed files
Comments generated: 8

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

…kel/azure-sdk-for-java into users/fabianm/test

Copilot

Copilot's findings

Files reviewed: 17/17 changed files
Comments generated: 3

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Copilot

Copilot's findings

Files reviewed: 17/17 changed files
Comments generated: 1

FabianMeiswinkel · 2026-06-19T11:47:25Z

/azp run java - cosmos – tests

azure-pipelines · 2026-06-19T11:47:32Z

No pipelines are associated with this pull request.

FabianMeiswinkel · 2026-06-19T11:47:35Z

/azp run java - cosmos - tests

azure-pipelines · 2026-06-19T11:48:05Z

Azure Pipelines successfully started running 1 pipeline(s).

FabianMeiswinkel · 2026-06-19T12:05:01Z

/azp run java - cosmos - tests

azure-pipelines · 2026-06-19T12:05:29Z

Azure Pipelines successfully started running 1 pipeline(s).

FabianMeiswinkel · 2026-06-19T15:06:26Z

/azp run java - cosmos - tests

azure-pipelines · 2026-06-19T15:07:01Z

Azure Pipelines successfully started running 1 pipeline(s).

FabianMeiswinkel · 2026-06-19T15:09:31Z

/azp run java - cosmos - tests

azure-pipelines · 2026-06-19T15:09:56Z

Azure Pipelines successfully started running 1 pipeline(s).

xinlian12

LGTM, thanks

…kel/azure-sdk-for-java into users/fabianm/test

FabianMeiswinkel · 2026-06-22T08:13:47Z

/azp run java - cosmos - tests

azure-pipelines · 2026-06-22T08:14:15Z

Azure Pipelines successfully started running 1 pipeline(s).

FabianMeiswinkel · 2026-06-22T09:18:53Z

/azp run java - cosmos - tests

azure-pipelines · 2026-06-22T09:19:28Z

Azure Pipelines successfully started running 1 pipeline(s).

FabianMeiswinkel · 2026-06-22T09:59:48Z

/azp run java - cosmos - tests

azure-pipelines · 2026-06-22T10:00:31Z

Azure Pipelines successfully started running 1 pipeline(s).

FabianMeiswinkel · 2026-06-22T11:57:55Z

/azp run java - cosmos - tests

azure-pipelines · 2026-06-22T11:58:26Z

Azure Pipelines successfully started running 1 pipeline(s).

Adding more scenario-based tests

50c10b1

Copilot AI review requested due to automatic review settings June 19, 2026 08:53

FabianMeiswinkel requested review from a team and kirankumarkolli as code owners June 19, 2026 08:53

github-actions Bot added the Cosmos label Jun 19, 2026

Copilot started reviewing on behalf of FabianMeiswinkel June 19, 2026 08:54 View session

Copilot AI reviewed Jun 19, 2026

View reviewed changes

Potential fix for pull request finding

716e229

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Copilot AI mentioned this pull request Jun 19, 2026

Fix fault injection and assertion issues in customer workflow tests (PR #49568 review) #49569

Draft

FabianMeiswinkel added 3 commits June 19, 2026 09:15

Update live-platform-matrix.json

e68f653

Merge branch 'users/fabianm/test' of https://github.com/FabianMeiswin…

74ade95

…kel/azure-sdk-for-java into users/fabianm/test

Addressing code review feedback

0b236f1

FabianMeiswinkel requested a review from Copilot June 19, 2026 11:14

Merge branch 'main' into users/fabianm/test

e3f3f23

Copilot started reviewing on behalf of FabianMeiswinkel June 19, 2026 11:15 View session

Copilot AI reviewed Jun 19, 2026

View reviewed changes

FabianMeiswinkel and others added 3 commits June 19, 2026 13:23

Potential fix for pull request finding

e01dbdb

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Register created items for cleanup in customer workflow create paths

837f3da

Update CustomerWorkflowHighE2ETimeoutTest.java

fb9e0db

FabianMeiswinkel requested a review from Copilot June 19, 2026 11:35

Copilot started reviewing on behalf of FabianMeiswinkel June 19, 2026 11:36 View session

Copilot AI reviewed Jun 19, 2026

View reviewed changes

Comment thread sdk/cosmos/azure-cosmos-tests/CUSTOMER_WORKFLOW_COVERAGE_MAP.md Outdated

Delete CUSTOMER_WORKFLOW_COVERAGE_MAP.md

a37f517

Refactoring customer tests into separate pipelines

9f6c651

FabianMeiswinkel marked this pull request as draft June 19, 2026 13:23

FabianMeiswinkel added 4 commits June 19, 2026 14:05

Fix single master tests

b946a65

Update CustomerWorkflowChangeFeedProcessorTest.java

c422d86

Update CustomerWorkflowTestBase.java

3ff626d

Merge branch 'main' into users/fabianm/test

fd17a99

FabianMeiswinkel marked this pull request as ready for review June 19, 2026 16:18

jeet1995 approved these changes Jun 19, 2026

View reviewed changes

xinlian12 approved these changes Jun 19, 2026

View reviewed changes

FabianMeiswinkel added 3 commits June 22, 2026 08:04

Fixing test flakiness due to async container creation

81f1de9

Merge branch 'users/fabianm/test' of https://github.com/FabianMeiswin…

f1d585e

…kel/azure-sdk-for-java into users/fabianm/test

Merge branch 'main' into users/fabianm/test

0bf70bd

Fixes test failures

f530460

FabianMeiswinkel added 2 commits June 22, 2026 09:56

Update CustomerWorkflowStoredProcedureTest.java

d88fe4a

Update PerPartitionAutomaticFailoverE2ETests.java

f79896e

Fixing tests

f122a12

Conversation

FabianMeiswinkel commented Jun 19, 2026

Description

All SDK Contribution checklist:

General Guidelines and Best Practices

Testing Guidelines

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Copilot's findings

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot's findings

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot's findings

Uh oh!

Uh oh!

FabianMeiswinkel commented Jun 19, 2026

Uh oh!

azure-pipelines Bot commented Jun 19, 2026

Uh oh!

FabianMeiswinkel commented Jun 19, 2026

Uh oh!

azure-pipelines Bot commented Jun 19, 2026

Uh oh!

FabianMeiswinkel commented Jun 19, 2026

Uh oh!

azure-pipelines Bot commented Jun 19, 2026

Uh oh!

FabianMeiswinkel commented Jun 19, 2026

Uh oh!

azure-pipelines Bot commented Jun 19, 2026

Uh oh!

FabianMeiswinkel commented Jun 19, 2026

Uh oh!

azure-pipelines Bot commented Jun 19, 2026

Uh oh!

xinlian12 left a comment

Choose a reason for hiding this comment

Uh oh!

FabianMeiswinkel commented Jun 22, 2026

Uh oh!

azure-pipelines Bot commented Jun 22, 2026

Uh oh!

FabianMeiswinkel commented Jun 22, 2026

Uh oh!

azure-pipelines Bot commented Jun 22, 2026

Uh oh!

FabianMeiswinkel commented Jun 22, 2026

Uh oh!

azure-pipelines Bot commented Jun 22, 2026

Uh oh!

FabianMeiswinkel commented Jun 22, 2026

Uh oh!

azure-pipelines Bot commented Jun 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants